Method and device for enhancing the recognition of speech among speech-impaired individuals

ABSTRACT

A method and a system is disclosed that provide means to enable individuals with speech, language and reading based communication disabilities, due to a temporal processing problem, to improve their temporal processing abilities as well as their communication abilities. The method and system include provisions to elongate portions of phonemes that have brief and/or rapidly changing acoustic spectra, such as occur in the stop consonants b and d in the phonemes /ba/ and /da/, as well as reduce the duration of the steady state portion of the syllable. In addition, some emphasis is added to the rapidly changing segments of these phonemes. Additionally, the disclosure includes method for and computer software to modify fluent speech to make the modified speech better recognizable by communicatively impaired individuals. Finally, the disclosure includes method for and computer software to train temporal processing abilities, specifically speed and precision of temporal integration, sequencing and serial memory.

This is a division of application Ser. No. 08/858,961, filed May 20,1997, now U.S. Pat. No. 5,813,862, which is a continuation ofapplication Ser. No. 08/351,803, filed Dec. 8, 1994, now abandoned.

BACKGROUND OF THE INVENTION

This invention relates to the modification of audio sounds, inparticular, speech sounds, to enable individuals who are impaired withspeech and language-based learning disabilities (L/LDs) due to atemporal processing problem to improve their speech reception, speechproduction, language comprehension and reading abilities. In addition,it includes training methods to help individuals with speech andlanguage-based learning disabilities to overcome their temporalprocessing deficits so that they can recognize basic speech elements andnormal connected speech with higher accuracy and greaterintelligibility. In addition, it includes a training method to helpnormal individuals in the improvement of their speech receptioncapabilities, for either their native language, or for foreign languagetraining.

Recent studies have shown that specific language impaired (SLI) andspecifically reading impaired (dyslexic) individuals have an inabilityto recognize and distinguish between certain consonants andconsonant-vowel combinations in natural speech. They also havedifficulties in understanding written speech that appears to result fromtheir problems in understanding aural speech. This difficulty with auralspeech perception results in a delayed and usually defective developmentof reading skills. Studies have shown that these problems in speechreception and reading acquisition are not the result of peripheralhearing or visual deficits, but rather are due to an inability of thereceptive and cognitive powers of the brain to correctly identify therapidly changing components of speech. For example, L/LD individualshave difficulty correctly identifying the rather short consonant sounds(a few tens of milliseconds long) or to reliably separate them fromassociated longer vowel sounds. Consequently the individuals are unableto generate a reliable representation of the fundamental phoneticelements of the native language in their brains. The result is that theimpaired individual not only has difficulty correctly identifying theunique sounds of spoken words and strings of connected speech, but alsooften has associated difficulties in learning to accurately articulatespeech. In addition, the impaired individual may have limited cognitiveabilities that rely on accurately recognizing words and long speechstrings, and limited abilities in cognitively associating written speechwith their brain's poor representations of aural speech.

In particular, consonant sounds generally have a frequency modulatedcomponent such that the sound frequency may rise or fall, or beinterrupted by pauses that last for less than 25 milliseconds to morethan 80 milliseconds. This rising or falling sound frequency or briefinterruptions of the consonant sounds are followed or preceded by avowel sound, which has a relatively constant or more slowly changingspectral content, and which usually extends over a period of from manytens of milliseconds up to several hundred milliseconds. The majority ofindividuals with L/LDs (dysphasia or dyslexia) cannot distinguishbetween the consonant-vowel combinations (for example, /ba/ and /da/, or/ab/ and /ad/) when the frequency modulated components of the consonants/b/ and /d/ are of normal duration (for example, less than 60 to 80milliseconds long).

The basic temporal processing deficit in L/LDs is also reliablydemonstrated by testing a dysphasic and/or dyslexic individual's abilityto identify sounds that are presented in rapid succession, as commonlyoccurs for successive phonetic elements in normal speech reception. Forexample, an L/LD child or adult commonly cannot correctly identify theorder of presentation of two different, successive vowel-like stimulithat are each 50 milliseconds in duration unless they are separated intime by more than 100 milliseconds, and often by more than severalhundred milliseconds. By contrast, a normal individual can identify thesequence order of presentation of such stimuli when they are immediatelysuccessive, that is, with no intervening interstimulus time gap.

The result of this fundamental problem in the reception ofshort-duration and fast successive components of speech is readilyapparent in the school system, where individuals diagnosed withdysphasia and dyslexia with this temporal processing difficulty will runtwo-to-four, and maybe more, years behind their peers in scholasticachievement. The result is that L/LDs commonly require additionalspecialized training, with great emphasis on speech recognition andspeech production. Dyslexics similarly receive special training to helpthem learn to read. Special speech reception, speech production andreading instruction continues generally throughout the elementary andsecondary school educations for many of these individuals if theresources are available. The impairment can often lead to a truncationin education, and commonly results in impairment for life. However, somesuccess is shown for special training.

Initially, failure of identification of consonant-vowel combinationssuch as /ba/ and /da/ with short duration consonant frequencymodulations of less than 60 milliseconds, or failure to identify thetemporal order of simple acoustic stimuli unless they are separated by150 or more milliseconds, has established a means of identifying L/LDswith this temporal processing deficit. However, no prior trainingstrategy has shown consistently positive results in overcoming thetemporal processing deficits that underlie L/LDs. Overcoming thistemporal processing deficit should result in a more useful and normallife for individuals with this affliction.

Recent studies have shown that these speech and language-based learningdisabilities are seated in defective temporal processing of sensoryinformation by the brain. Moreover, they have shown that temporalprocessing abilities are subject to strong learning effects in normalindividuals. The basic processes underlying this temporal processlearning are increasingly better understood.

In addition to L/LDs, brain damaged individuals have shown similarsymptoms. In particular, individuals who have suffered strokes orotherwise damaged portions of their language-dominant cerebralhemispheres commonly lose the ability to discriminate between normalconsonant sounds and show temporal processing deficits that are verysimilar to those in L/LD individuals. As with L/LD individuals, theseaphasic individuals can also correctly identify speech elements whenthey are presented to the patient in a slowed-down form.

Aged individuals also show a progressive deterioration in their temporalprocessing abilities, as judged by these same tests. This deteriorationcontributes to a cognitive-based deficit that affects their speechreception and general cognitive abilities.

The reception of, or learning of, a foreign language in an indigenousenvironment is difficult and sometimes almost insurmountable for normalindividuals because of the speed at which the language is spoken.Foreign languages are consequently learned by rote memorization andrepeated practice exercises, with the speed of talking increasedcommensurate with the ability to understand the spoken language. Thereis no set means for individuals learning a foreign language in theindigenous environment (that is, in the native country of the language)except by asking the foreign language speaker to "slow down" or to"repeat". Most of the problems in learning foreign languages in thisindigenous environment can be attributed to the lack of recognition inthe temporal processing of fast events in one's brain of the incomingspeech sounds.

While the phonemes of foreign languages differ in construction from theEnglish language, the principles behind all spoken languages remainconstant. That is, all languages can be broken down into fundamentalsound structures known as phonemes. It is the recognition of thesephonemes, such as the consonant-vowel syllables /ba/ and /da/ in theEnglish language, that form the basic building blocks that must belearned. As with the L/LD individual, the foreign language student doesnot recognize these phonemes reliably when they are presented at theirnormal element durations and normal element sequence rates by nativelanguage speakers. As with L/LDs, they can be accurately distinguishedfrom one another and can be correctly identified when the speech isartificially slowed down.

It is an object of this invention to provide a means for easierrecognition of phonemes and connected speech in L/LD individuals.

It is a further object of this invention to provide a training strategyfor rapidly and progressively improving the recognition of phonemes andconnected speech in L/LD individuals.

It is another object of this invention to employ training signals thatare more powerful than normal speech for generating changes in temporalprocessing by the brain achieved through learning exercises.

It is the further object of this invention to use a modified version ofthis training strategy as a method for screening human populations toidentify those individuals who would benefit from this invention.

It is also an object of this invention to provide phoneme and connectedspeech recognition and a training strategy for rapidly and progressivelyimproving the recognition of phonemes and connected speech inindividuals that have suffered brain damage to their dominantspeech-language hemisphere that has resulted in a temporal processingdeficit like that recorded in L/LDs.

It is a further object of this invention to provide phoneme andconnected speech recognition and a training strategy for rapidly andprogressively improving the recognition of phonemes and connected speechin individuals who have undergone age-related or disease-relateddeterioration of their temporal processing abilities for speech soundreception.

It is still a further object of this invention to provide easierrecognition of phonemes and connected speech in the learning of aforeign language.

It is still a further object of this invention to provide improvedtemporal processing of fast speech sounds in normally fluentindividuals, to improve their learning capabilities and their potentialcognitive achievements.

In one aspect of the invention, a method of increasing the ability of ahuman being to process aurally received signals is disclosed as amethod. The method consists of recording audio sounds in a computerizedsystem. The method includes a step of modifying the amplitudes andtimings of recordings of certain phonetic elements without changingtheir fundamental frequencies. Finally, the invention includesconverting the modified digital signals to analog aural signals forpresentation to the individuals.

In a second aspect of the invention, a method for increasing the abilityof a human being to distinguish and separate fast sequential aurallyreceived signals is disclosed as a method. The method consists ofcontrolling the sound output of fast phonetic or non-speech sounds incomputer-mounted games, at which the human being works to correctlyrecognize progressively faster sound presentations, or in which thehuman subject works to distinguish the time order of presentation or theseparate identities presented at progressively shorter durations, atprogressively faster rates, and with progressively longer and longersound element sequences.

The invention also encompasses a method for increasing the ability of ahuman being to recognize long, connected speech strings, and to rapidlyimprove their performances at related cognitive tasks. Speech sounds oftraining exercises designed for L/LD children are modified in acomputerized system, as above. All speech applied in training exercisesand on library materials are delivered in this modified form.

The invention also encompasses a computerized system for structuringrecorded audio information to enable speech and language impaired ornormal individuals to better understand spectro-temporally complex audiosounds. The system consists of a computer having input means forreceiving information including the processor means for manipulating thereceived information; storage means for storing unprocessed, receivedinformation and manipulated information; and output means responsive tothe processor means for presenting the manipulated information in a formunderstandable, and providing an effective learning signal, to a user.The computerized system also includes first program means for modifyingdigitally recorded audio sounds having a frequency range associatedtherewith to lengthen and to selectively amplify fast (primarilyconsonant) acoustic elements in speech without modifying the frequencyrange. Second program means are provided responsive to the receivedinformation and the processor means for storing the modified, digitallyrecorded speech sounds. Third program means responsive to the receivedinformation and to the processor are provided to direct the stored,modified, digitally recorded speech sounds to the output means.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a general diagram of a computer system that would be used toimplement remediation of auditory temporal processing deficits;

FIG. 2 is a spectrographic plot of a low to high frequency modulatedsweep (S1) and a high to low frequency modulated sweep (S2) separated byan interstimulus interval (ISI);

FIG. 3 is a depiction of a VDT with graphical icons and symbols thereon;

FIG. 4 is a graphical representation of a normal consonant-vowelcombination, plotting frequency against time;

FIG. 5 is a graphical representation of a time-expanded consonant-vowelcombination, plotting frequency against time;

FIGS. 6A, 6B, 6C and 6D are flow charts showing a computer program thatis used to develop the training program;

FIG. 7A is a graph of the results of the Tallal Repetition Test beforetraining of seven selected L/LD children;

FIG. 7B is a graph of the results of the Tallal Repetition Test aftertraining of seven selected L/LD children;

FIG. 8 is a graph of the results of the GFW test of auditorydiscrimination for the seven L/LD children showing the actual ageplotted against the age equivalent both for pretest, post-test and threemonths after the training session, where available;

FIG. 9 is a graph of the results of the TROG (Test for Reception ofGrammar) for the pretest and post-test times, showing the equivalent ageof the seven L/LD children;

FIG. 10 is a graph of the results of the Comprehensive LanguageEvaluation (CYCLE) test for the seven L/LD children for the pretest andpost-test times and, where available, three months after the training;

FIG. 11 is a graph of the Token Test (receptive language) for the sevenL/LD children for the pretest and post-test times and, where available,three months after the training;

FIG. 12 is a graph showing the improvement on the CYCLE test for L/LDchildren from age 4 to age 8 over a five year period and the improvementin the seven tested L/LD children over the four week training sessionusing this invention;

FIG. 13 is a graph showing the improvement on the Token Test for L/LDchildren from age 4 to age 8 over a five-year period and the improvementin the seven tested L/LD children over a four week training sessionusing this invention;

FIG. 14 is a graph showing the improvement on the GFW test for L/LDchildren from age 4 to age 8 over a five-year period and the improvementin the seven tested L/LD children over a four week training sessionusing this invention;

FIG. 15 is a graph of the improvement in the CELF test (sentenceimitation) for the seven tested L/LD children at pretest, post-test and,where available, three months after training;

FIG. 16 is a graph of the results of the memory for sentences for theseven L/LD children at pretest, post-test and, where available, threemonths after training;

FIG. 17 is a graph showing the results of the Goldman-Fristoearticulation test on the seven L/LD children at pretest, post-test and,where available, three months after training.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring to FIG. 1, a computer system appropriate for use in thepresent invention for enhancing the recognition of speech amongspeech-impaired individuals (L/LDs) is shown. The computer system 10consists of a processor 12 made up of a conventional type system such asavailable from Apple or one of the IBM or IBM-compatible clones. Theprocessor 12 includes a CD-ROM capability 14 and a conventional floppydisk capability 16. Appropriate software is provided in the processor 12to support not only the CD-ROM but also the floppy disk capability.

In addition to the processor 12, a conventional visual display tube orVDT 18 is included, which may or may not be capable of displaying color,although the color format enhances two aspects of this invention,namely, the game-playing and training exercise aspects. Additional VDT'smay also be included.

A speaker 20 is provided which is supported by appropriate software inthe processor 12. In addition to the speaker 20 are headsets 22 whichcan privatize the invention to an individual person(s).

Finally, to make the system usable to the subject or patient, storage 24is provided to store the appropriate programs and also the appropriatedata that is utilized. Storage 24 is responsive to processor 12, as arethe VDT 18, the speaker 20, and headsets 22. Storage 24 can also be usedto accumulate progress reports and generate statistics on the individualuser(s).

The VDT 18, the speaker 20, and headsets 22 are hereinafter referred toas the "output" when referred to generally in the system. The inputdevices include a keyboard 26, which may or may not be utilized by thepatient, and a mouse 28, which is appropriate for use in the entireinvention. A touch sensitive screen 33, can be used in lieu of the mouseand is particularly useful with young children. Finally, an inputmicrophone 30 may be provided to record various oral sounds in order toassess speech production performance by the L/LD individual or to assistin the development of additional verbal program material.

A printer 32 may also be included to provide output on the progress ofthe patient being trained in this system.

While not necessarily a part of the system, the processor 12 may be aperipheral unit to a host computer 34 in a network of computers. Hostcomputer 34 could control several processors 12 with the appropriateperipherals, as shown in FIG. 1. Other systems such as 10(b), (c), (d),and (e) would, in effect, be essentially the same as the system 10.Further modification may place the printer, for example, under directcontrol of the host computer so it would not be necessary to have aprinter at each remote station.

In addition to the computer hardware set forth above, in order tooperate this system and enhance the training program, certain audio taperecorders are required for use by the student in running this system.These tape recorders are shown as 36 and 38 in FIG. 1. They are attachedto host computer 34 for convenience's sake; however, it should beunderstood that the tape recorders can be independent and in thatinstance would include their own headsets 36(a) and 38(a). In addition,microphones 36(b) and 38(b) would also be included with the taperecorders. Thus tape recorders 36 and 38 can not only provide audiosignals to the headsets 36(a) and 38(a), but also can receive inputthrough microphones 36(b) and 38(b) in the conventional manner. The taperecorders are shown associated with host computer 34 for convenience'ssake only; however, they can be stand-alone for the various tests, gamesand training regimes herein.

The purpose of the computer system 10 is to develop appropriate trainingprograms and to provide a medium for patients to practice their languageskills, and, in particular, to overcome handicaps or deficits intemporal processing that have resulted in impairment of speech receptionand/or production, or to improve their normal temporal processingcapabilities. In particular, computer system 10 is utilized to provide amedium for L/LD or normal subjects including school-aged children, agedindividuals and foreign language students, or aphasic patients todistinguish between differing sounds. For example, in a program recordedon CD-ROM, a game is presented to young children between the ages of 5and 10 wherein the computer not only provides the training but alsoprovides rewards to the subject students. In the example used in theinvention, a sequence is developed around a circus game, complete withmusic and video signals showing clowns, animals, and the like, asbackground to audio signals.

FIG. 2 is an example of the audio sounds presented in one of theelementary games employed in training. In FIG. 2, two differing audiosignals, one with an increasing frequency S₁ and one with a decreasingfrequency S₂, are presented to a student with an inter-stimulus interval(ISI) determined by the program. The signals S₁ and S₂, as shown byexample in FIG. 2, are varied in the sequence; that is, the increasingsignal S₁ may be sounded first, followed by the decreasing signal S₂ inthe first presentation, while in the second, the decreasing signal S₂ isfollowed by the increasing signal S₁. The student is asked to respondutilizing the mouse 28 by moving a cursor between two boxes 40 and 42displayed on VDT 18 (see FIG. 3).

For example, the increasing sound would always be the left box 40 andthe decreasing sound would always be the right box 42. If the decreasingsound came first, the student would move the cursor to the box on theright and "click" or touch on the right box. If the increasing soundcame first, then the student would click or touch on the left box.Correct responses result in a display on the screen of a clown dancing,some music playing, or some other appropriate visual reward.

A sequence of several correct responses, for example, three, results ina decrease in the interstimulus interval, for example, from 200 to 195milliseconds. Continued improvement has been shown in presenting such aproblem to a L/LD child over a period of four to six weeks. More will besaid about this in the results of the experiments conducted, of which adescription is provided below.

Once the patient completes a certain amount of auditory sequencetraining, the patient would be shifted to a different type of display onthe VDT, within the format of the same or a different game. Inparticular, stop consonant syllables for testing and training phonemeidentification such as the stop consonants /ba/ and /da/ would be usedas target signals in a game. The purpose of this portion of the trainingis to overcome a common failing in dysphasic children in distinguishingbetween various consonants. For example, the two consonants /ba/ and/da/ are quite often confused in this situation.

Initially, the patient is tested in a manner similar to that set forthabove. In the phoneme identification training the patient is required toindicate if the target phoneme was presented first or second in thesequence. For example, by first distinguishing between the sounds /ba/and /da/ with a long interstimuli interval of upwards of 500milliseconds. During training, the ISI is decreased as the patientprogresses in recognizing the sounds /ba/ and /da/ and other stopconsonants. However, this system envisions taking the phoneme /ba/ orthe phoneme /da/, for example, and prolonging the fast-acting or stopconsonants, /b/ and /d/, respectively. In a normal speech pattern, theconsonants /b/ and /d/ in these two phonemes occur in approximately 40to 50 milliseconds, while the following vowel sound of the letter "a"may be five or six times as long. These sounds are representedgraphically in FIG. 4. In order to improve the patient's recognition ofthese sounds, it has been found that if the consonant portion of thesound, that is, the /b/ or the /d/, is prolonged or doubled toapproximately 80 milliseconds, then the patient can consistently andaccurately identify the phoneme. The initial lack of recognition isattributable in the impairment in the temporal processing in theindividuals with the learning disability.

In FIG. 5, this prolonged consonant sound is shown with a commensuratedecrease in the duration of the vowel sound. The decrease in the vowelsound need not be accomplished in the training phase; however, as willbe explained subsequently, when it is necessary to "rescript" anexisting recording, be it oral or video, for use by individuals with alearning disability, it is appropriate to truncate the vowel sounds in"real time" so that the elapsed time of the recording is not changed.Hence, any extension of a portion of a phoneme must be accompanied by anequivalent reduction in another portion of the spoken word. Thus, FIG. 5represents such a reduction, where the elapsed time of the modifiedphoneme (240 ms) is the same as the unmodified phoneme shown in FIG. 4.Careful editing of the original audio information may permit sometruncation of the silent periods between words or sentences.

The testing of the /ba/ and /da/ identification, or an equivalent stopconsonant identification, is accomplished in a similar manner as setforth above, and can be accompanied by a circus-like game or othercontext to make it more attractive to young children. As the subjectachieves improved ability to correctly identify phonemes with consonantsounds of a given duration, the task can be made more difficult to driveimprovements in performance, for example by shortening the stopconsonant sound duration from 80 to 70 msec, and so forth.

In order to provide continued interest, variety, and additional exposureto modified speech training material, it has been found thatcommercially available "Books on Tape" and video games, in particularinteractive CD-ROMs that are rich in speech content, becomeunderstandable to learning impaired individuals if the verbal materialis modified to lengthen the stop consonants, then matching the modifiedsound with the original video material. Lengthening of 50% to 100% hasbeen found to be appropriate. If there are accompanying video oranimated graphics, as is the case with video games, then the visualmaterial may need modification also, or the sound may need compensatingcompression to fit a real-time situation. This same procedure can beused in the teaching of a foreign language where the fast consonants areinitially lengthened appropriately.

In addition to lengthening, it is also important to emphasize the fastor stop consonants by increasing the energy thereof by, for example, 20dB. This increases the relative salience for learning these criticalspeech elements that are so poorly understood by L/LD and other temporalprocessing impaired humans. The time envelope of this differentialamplification of fast speech elements is an important second variable ofthis fast or stop consonant element emphasis.

Having briefly explained the nature of the tests, the nature of thetraining with respect to this invention, and the necessary changes toexisting material, it is appropriate to pass on to software that may beused to develop content test and training material, to rescript "Bookson Tape", or to modify and rescript existing video games.

Reference should be made to FIGS. 6A, 6B, 6C and 6D. It is to beunderstood that the flow charts in FIGS. 6A, 6B, 6C and 6D arerepresentative only, and could be done in separate increments ratherthan a single program as illustrated herein.

Referring to FIG. 6A, the user is first faced with a decision whethertraining material is to be developed (RECORD ONLY--decision block 50),"Books on Tape" are to be modified for use by individuals with learningdisabilities (VOICE ONLY--decision block 52), or whether a CD-ROM with avideo game is to be modified (MULTIMEDIA--decision block 54). Thetraining material, as noted above, include the stop consonant phonemes,for example, /ba/ and /da/. The "Books on Tape" are clear from thetextual point of view. However, to clarify, it has been foundappropriate to modify "Books on Tape" such that an individual withlearning disabilities can understand the modified spoken words which,without modification, would not be as intelligible to that individual.CD-ROM or multimedia, of course, includes video games.

Assuming, in this instance, that test materials that are to be developedare selected in the decision block 50 illustrated in FIG. 6A, one isthen directed to FIG. 6B, with the first step being to record thematerial on a digital audiotape recorder or DAT, the purpose being thatspeech modification cannot be readily accomplished in an analog mode.

The next step is to convert the digital audiotape to a special formatwhich is adaptable to elongation of certain portions of the speech. Theformat used in developing this invention is the audio information fileformat or AIFF which was developed by Apple Computer and is supported bySilicon Graphics. This format supports a 16 bit file with a samplingrate of 22 kHz although other sampling rates and integer representationscan be used (e.g. 8 bits, 11 kHz).

The next step indicated in operation block 60 is to segment the soundstream into subfiles that reduce computer memory requirements. Hereagain, in this invention the sound stream was segmented into 30 secondsubfiles but other lengths would be equally appropriate depending onavailable processor memory.

The user is then faced with the decision of whether to only modify thetime scale, only modify the emphasis in the voice stream, or to do both.In the instance of modifying the time scale, one is directed to FIG. 6Cfor further direction. It is to be understood at this point thatmodifying the time scale is to elongate the stop consonants and otherfast speech elements and, in the instance of the real time environment,to shorten the following or leading vowel sounds, or in other instances,to decrease the "dead" time that occurs between sentences and betweenwords.

Referring now to FIG. 6C and operation block 66 the time scalemodification first converts the incoming signal by means of a short-termFast Fourier transform consisting of 128 simultaneous channels eachrepresented by a complex spectra. The complex spectra in mathematicalterms is a series of complex numbers each having a real and imaginarypart, well-known in the electronic and mathematical fields, which isrepresented in the Fourier transform.

This complex spectra is then converted to frequency and amplitude asindicated in operation block 68. Once the spectra is converted to thefrequency and amplitude representation indicated in block 68, thespectra is interpolated between successive short-term spectral timeslices in order to create the desired expansion in the time domain ineach frequency channel. That is, the given frequency and amplitude for aparticular instance of time is compared to the two adjacent frequenciesand amplitudes as determined by the Fast Fourier transform and a smoothtransition in amplitude is created for each frequency channel. In thisinstance, the sound is literally spread out and additional informationis inserted (interpolated) between each of the representations of thesound spectra in the Fourier transform. It has been found that a 50%increase in time is commonly adequate to accomplish the necessary changein the stop consonants coming through to the individual with thelearning disability. Thus, block 70 indicated in the interpolation ofthe spectra results in expansion of the time spectra. In like manner, ifit becomes necessary to compress the time spectra, this can also beaccomplished in this phase by deleting a particular set of Fouriertransforms and then interpolating between the remaining adjacenttransforms to smooth the curve.

After interpolation, an additive synthesis is accomplished as indicatedin operation block 72. In short, the additive synthesis step convertseach of the frequencies and amplitudes back into the time domain or, ineffect, performs an inverse Fast Fourier transform. Following theadditive synthesis, each channel is put back into a time domain in aform such as existed prior to operation block 66. Following operationblock 72, the program returns to the mainstream as shown in FIG. 6B todetermine if emphasis should be incorporated into the speech. (See thedecision block 64 in FIG. 6B.) Comparison of FIG. 4 and FIG. 5 shows aschematic representation of the elongation.

As noted above, two things can be accomplished in preparing speech andphonemes for use by individuals with a learning disability. One is tomodify the time scale of the speech, which has just been discussed, andthe second is to emphasize, by increasing the energy content, thoseportions of speech that contain rapid temporal changes in frequencycontent. It has been found that if the stop consonants such as the /ba/or /da/ representations noted above are modified so that the energy ofthe consonant portion of the phoneme is increased, then improvement inthe learning capability of an individual with a learning disability isalso seen. In this instance, if emphasis is desired, dynamic emphasis isundertaken, as illustrated in FIG. 6C starting with multichannelbandpass filtering to create a number of separate frequency channels asindicated in operation block 74. In the present instance, the number offrequency channels has been selected at 40; however, a different numbercould be selected.

Filtering by Butterworth filters by the fourth order creates the 40channels across a time interval of the previously segmented sound stream(see operation block 60), of which each segment consists of 30 seconds.In operation block 76 (FIG. 6C), all of the channels are filteredthrough a 1-6 kHz filter and provide for a 10-20 dB emphasis forfrequencies that fall into the flat bandpass response betweenapproximately 2-4 kHz. Subsequently, a gross amplitude envelope isextracted and defined for each of the 40 frequency band channels overeach 30 second subfile as indicated in operation block 78. For thoseenvelopes that show a rapid change (see operation block 80, FIG. 6C),the extracted envelopes are emphasized 10-20 db. It is pointed out thatfor consonants such as the examples of the /ba/ and /da/, the /b/ in the/ba/ and the /d/ in the /da/ as noted above will always exhibit a rapidchange in the envelope of the frequency channels as opposed to therelatively constant envelope that is seen in a vowel block. The specificfrequency channels for which a particular degree of emphasis is applied,the gross amplitude envelope frequency thresholds that marks theoccurrence of a rapid change in the speech signal that triggersemphasis, and the degree of emphasis as a function of the rapidity(frequency) of gross amplitude envelope changes are all separatelyselectable. In the present application, a uniform emphasis of 10-20 dBwas applied for the 2 to 4 kHz channels with gross amplitude envelopeswith frequency components ranging from about 3 to about 45 Hz fornatural speech, and from about 2 to about 30 Hz for 150% time expandedspeech. Emphasis was progressively reduced as described above below 2kHz, and above 4 kHz.

In order to create a new sub-band envelope as indicated in operationblock 82, each channel is then modified by dividing by the informationobtained in the initial gross amplitude envelope for each channeldetermined in block 78. For each channel, the resulting information ismultiplied by the channel envelope developed in operation block 80. Thisproduces an envelope modification function for each channel. Eachchannel envelope modification function is then multiplied channel bychannel with the original channel signals defined in operation block 74.Additive synthesis is performed in operation block 84 such that theplurality of frequency bandpass channels can be converted via additionto a single channel across the entire frequency range contained in thedigitized speech. Once the single channel is arrived at, then theprogram returns to FIG. 6B at connector E so that the sound segmentsdetermined in operation block 60 can be reassembled as shown inoperation block 86.

Referring back to FIG. 6A, if a multimedia CD-ROM is present, then indecision block 54, it branches to an operation block 88 to identify theaudio format in which the CD-ROM has been recorded. For example, CD-ROMsmay come in any one of a number of formats, some of them being the SoundResource Format, a Quick Time Format, a Paco Format, or the AIFF formatwhich is used in the subsequent steps in this program. If the soundsource is in some other format than AIFF, then it is converted to theAIFF format for operation in this program. (It should be noted that anequivalent program could be written to operate on other sound sources,just as well as the AIFF.) Further, the information must be converted todigital if it is analog. This is accomplished in operation block 88.

For convenience, a flag is set at this point indicating that the programis in a CD-ROM mode. The CD-ROM flag is set in operation block 90. Theprogram then branches at connector F to the mainstream shown in FIG. 6B.

With the CD-ROM, the information is processed as indicated above,modifying the time scale and, if need be, applying the emphasis usingexactly the same steps indicated above.

The sound is reassembled in operation block 86. At that point, theCD-ROM flag is checked and, if on, the program shifts to FIG. 6D,operation block 92, where the modified sound is converted back into theoriginal audio format determined in operation block 88 (see FIG. 6A).The sound is then reinserted into the sources at the proper locations asindicated at operation block 94. That is, the video and/or animatedgraphics portions of the CD-ROM are synchronized with the modified soundportions.

Concurrently, the video and or animated graphics playback speeds areadjusted to match the modified speech as indicated in block 96.Alternatively, the modified speech can be placed in a real time mode, asindicated above, where the elapsed time of the speech in the modifiedversion is identical to the elapsed time in the speech in the originalversion so that the motion and the like on the CD-ROM correspond to theoriginal version. Finally, a master CD-ROM is generated, as indicated inoperation block 98, and the program is completed.

Referring back to FIG. 6B, the output data is then converted back intoanalog tape for the "Books on Tape" or for tests that can beadministered using analog tape. Alternatively versions of the tests andassociated test graphics can also be placed on the computer in digitalform where they can be administered under computer control.

It should be noted that the "Books on Tape" is a voice only situation,and corresponds exactly to the test material version once the "Books onTape" is converted to a digital audio tape indicated in block 100.Following that conversion to digital audio tape, the program follows theexact same steps as in the test material.

Operation of the preferred embodiment is exemplified in the followingexperiment, which used the aforedescribed material.

Experiment--Procedures Used for Remediation Studies Subjects

Seven children with specific developmental language based learningdisabilities (L/LD) participated as subjects in this study. L/LDchildren were selected who were without other primary sensory, motor,cognitive, emotional or neurological impairments, and who met thefollowing criteria: 1) nonverbal performance IQ of 85 or better on theWeschler Intelligence Scale appropriate for age; 2) at least 1.5standard deviations below the mean "predicted" achievement level on theage appropriate form of the Test of Language Development; 3) at least1.5 standard deviations below the mean for age and gender on the TallalTest of Auditory Processing; 4) normal hearing acuity, no motorhandicaps, no oral, motor or structural impairment effecting nonspeechmovements of the articulators; 5) no obvious signs of infantile autism,emotional difficulties or evidence of frank neurological disorders(seizures, hemiplegia etc.).

Schedule

The remediation study lasted six weeks with children receiving directtraining 3 hours per day, 5 days a week at the Rutgers UniversityExperimental Summer School, and doing 1 to 2 hours of homework per day 7days a week.

In weeks 1 "Benchmark" testing was done to determine each child's entrylevel ability on a series of standardized speech and language measuresas well as their temporal processing thresholds. Standardized speech andlanguage tests were tape recorded, but otherwise given according tostandardized format. If more than one form of the test was available,form A was always used. No acoustic modifications were used whenrecording or giving the "Benchmark" tests. Test performance was scoredusing standardized procedures according to the test manuals. In additionto these standardized speech and language measures, the Tallal Test ofAuditory Processing (Repetition Test) was given to establish temporalthresholds. Results of week 1 testing will be referred to as "pretest".In week 6, all of the "benchmark" measures were repeated. The results ofweek 6 testing will be referred to as "post test".

In weeks 2 through 5 remediation training was given. Each child rotatedthrough a series of 20 minute training sessions separated by two 20minute break/snack sessions. Sessions were scheduled in such a way as toassure that each child completed each 20 minute computer game session(sessions A, B) and story on tape or CD-ROM (session C) once a day. Eachof the receptive phonology and grammar sessions (sessions D, E, F, G andK) were given twice a week. The expressive speech and language sessions(session H, I and J), in which the children had to repeat verbatim whatthey heard in a series of real or nonsense syllables, words andsentences, were given once a week.

For the training sessions C through K above, materials were taperecorded or obtained directly from CD-ROM disks and subsequently theaudio signal was computer modified using the expansion and emphasisprocesses described in the invention. These modified tapes were thenused in all subsequent training sessions during weeks 2 through 5 bothduring the laboratory and homework sessions. For those training sessionsthat were using materials from standardized speech and language tests, acompletely different form of each test (form B) was constructed toassure that specific vocabulary and test items were not being trained.Rather, the training aimed to broadly remediate perception andproduction of phonological contrasts, knowledge of syntactic andmorphological rules, and verbal memory skills.

For receptive phonology and grammar sessions (session D, E, F, G and K)each processed speech command was presented and subjects were requiredto select one of several pictures which best represented the speechcommand (for example "Point to the book that is under the table"). Oncethe subject had responded, the experimenter indicated whether theresponse was correct with a "thumbs-up" or "thumbs-down" gesture.Regardless of whether the response was correct or incorrect theexperimenter would point to the correct picture and also draw thechild's attention to the portion of the picture which held the mostsalient information. Next, the same command was presented again so thatthe subject could hear it this time with advanced knowledge of thecorrect response. For expressive speech and language sessions (sessionsH, I, J) processed material was presented for verbatim response with nofeedback or repetition.

Each child in the study completed the following schedule. Benchmarkmeasures (unprocessed speech using form A) were given in week 1(pretest). Training sessions were given in weeks 2 through 5 whichincluded computerized training games, and exercises using processedspeech and language materials (form B), and processed storybooks andinteractional educational materials on tape and/or CD-ROM. Benchmarkmeasures were given again in week 6 using unprocessed speech form A. Nospecialized training was given in week 7 through 11 (although thechildren returned to their regular school classes during this time. Thechildren were brought back to the laboratory in week 12 at which timethe benchmark measures were given once again. Week 12 testing resultswill be referred to as "post-post testing".

Methods

Benchmark Measures--Given in Weeks 1 (Pretest), 6 (Post Test) and 12(Post-Post Test)

Tallal Test of Auditory Processing--(The Repetition Test, Tallal, 1980):In the Repetition Test, two different stimuli (stimulus 1 and stimulus2) are used in combination. Using operant conditioning procedures,subjects are trained to "repeat" each stimulus they perceive by pressingeither of two identical keys on a response panel, which are mounted oneabove the other. For trials that are comprised of more than onestimulus, subjects are required to "repeat" the pattern by pressing theappropriate response keys in the order of stimulus presentation. TheRepetition Test is comprised of a hierarchical set of subtests thatbuild successively one upon the other. These subtests allow for theinvestigation of 1) detection, 2) association/discrimination, 3)sequencing, 4) processing rate and 5) serial memory. Beginning with theDetection subtest, subjects must reach a set criterion of correctresponses in order to progress to the next highest subtest. A score isreceived for each subtest. In addition a composite score for performanceon all subtests completed is given.

1). Detection--The subject is initially presented with a single item(stimulus 1). The Experimenter (E) demonstrates that the bottom key onthe response panel is to be pressed every time stimulus 1 is presented.The subject is encouraged to respond and training continues until fivecorrect responses to this stimulus have been made. Next, the sameprocedure is repeated with the other stimulus (stimulus 2) and thesubject is trained to press the top key of the response panel. Thenumber of trials correct is recorded for stimulus 1 and stimulus 2.

2) Association/Discrimination--Stimulus 1 and stimulus 2 are presentedone at a time, in random order. The subject is trained to press thebottom key each time stimulus 1 is presented and the top key each timestimulus 2 is presented. Stimulus 1 and 2 continue to be presented oneat a time in random order, with immediate correction of errors, until acriterion of 10 out of 12 correct responses in a series of 16consecutive stimuli (P<0.002, Binomial Test--Siegel, 1956) is reached.This very strict criterion is employed as it is imperative to ensurethat each subject can discriminate between stimulus 1 and 2, and hasfully established the correct association to each of the two stimulibefore proceeding to the more complex tasks, which utilize combinationsof these two stimuli. The number of trials to criterion and number ofcorrect responses are recorded. If after twenty-four trials, a subjectfails to reach criterion, the test is terminated at this point.

3.) Sequencing--Subjects that reach the criterion on the Associationsubtest, are next trained to respond to stimulus 1 and 2, presentedsequentially, with a 500 msec inter-stimulus-interval (ISI). The fourpossible stimulus patterns (1-1, 2-2, 2-1, 1-2) are presented in randomorder. The subject is required to wait until both stimuli have beenpresented and then press the correct response keys in the same order asthe stimuli are presented. For each subject the method is demonstratedfour times by the E. Then, 12 test trials are given. During thesetrials, up to three errors are corrected. The score on this subtest isthe total number of correct responses.

4) Processing Rate--The same series of two-element stimulus pairs aswere used in the Sequencing subtest are presented again. However, inthis subtest the duration of the ISI is progressively reduced using anup/down staircase procedure to establish a psychophysical threshold,based on subject's responses. Subjects are given two practice trialsbefore beginning this subtest.

5) Serial Memory--Slow ISIs (500 msec)--In this subtest, the same twostimulus elements will be used and the procedure will be the same as forthe previous sequencing subtest, except that the number of elements inthe stimulus patterns will be increased. These stimulus patterns willconsist of 3, 4, 5, 6, or 7 elements, composed of random combinations ofthe two stimulus elements. The subject is given a stimulus patternincorporating 3 stimulus elements (i.e., 1-1-2). The experimenter willdemonstrate that the response panels are to be pressed 3 times in thecorresponding order in which the stimulus elements occurred. Then,additional 3-element patterns will be presented and the subject will beinstructed to wait until the entire pattern has been presented and thenmake the appropriate response to that stimulus pattern. Each subjectwill be given five, 3-element patterns. If the subject respondscorrectly to three of the five patterns, the same procedure will berepeated with the next highest pattern length. If, however, the subjectfails to respond correctly to three out of five patterns at any patternlength, this subtest will be terminated. A single score for the SerialMemory-Slow subtest will be obtained. This score will be made up of thetotal number of stimulus elements responded to correctly out of 125(5×3's+5×4's+5×5's+5×6's+5×7's).

6). Serial Memory--fast ISIs (10, 70 msec)--For each stimulus length onwhich a subject responded correctly on three of five trials on theSerial Memory subtest with 500 msec ISIs, five additional trials will begiven with ISIs of 10 and 70 msec. A single Serial Memory-Fast subtestscore, combining all of the fast ISI trials, will be calculatedsimilarly to the slow ISIs.

In this study the Repetition Test was given with nonverbal auditorystimuli. Two different computer generated complex tones were used.Stimulus 1 had a fundamental frequency of 100 Hz and Stimulus 2 afundamental frequency of 300 Hz. Thresholds were established for eachsubject with tone durations of 150 msec, 75 msec, 40 msec and 17 msec,respectively. The Serial Memory subtests were not given.

The test is administered via computer. Computer software insuresstandardization of stimulus presentation, response recording and scoringand threshold tracking. Auditory stimuli are presented binaurally overearphones at a constant super-threshold level (approximately 65 dBhearing level).

Receptive Language Tests

Token Test (DiSimoni, 1978): The Token Test was originally developed toassess receptive language deficits in adult aphasia. The test has beenused extensively in research and has proved highly sensitive toreceptive language impairments in both children and adults, and also tocontinued difficulties in adults with a documented childhood history ofL/LD. The Token Test utilizes large and small circles and squares offive colors. The test comprises four subtests of increasing memory loadand a fifth subtest with increased grammatical complexity. Afterdetermining that each subject is familiar with the test vocabulary, thetest begins with simple commands (i.e., touch the red circle). Eachsubsequent subtest gives commands with increasing memory load (i.e.,touch the big red circle; touch the red circle and the yellow square).In part 5, the commands add grammatical complexity (i.e., instead of theblue square pick up the white circle, etc.). The number of correctresponses on each subtest, as well as the total correct composite scoreare recorded.

Bishop Test for Reception of Grammar (TROG) (Bishop (1979): Bishop hasspent many years developing TROG, a rapid measure for assessingreceptive language. This measure has proven highly sensitive inassessing receptive language disorders in children, as well as in adultaphasics. TROG is an individually administered multiple choice testdesigned to assess understanding of grammatical contrasts in English.The test consists of 80 items. In each item the subject is required toselect from a 4-choice array the picture that corresponds to a phrase orsentence spoken by the E. The test is divided into blocks of four items,each block testing understanding of a specific type of morphological orsyntactic contrast. The test is scored in terms of the number of blockspassed, with the criterion for a pass being that all four items in thatblock are responded to correctly. Contrasts are arranged in order ofincreasing difficulty. Testing is started at a baseline of 5 consecutiveblocks correct, and is discontinued when five consecutive blocks havebeen failed. The test has been standardized on over 2,000 Britishchildren aged from 4 to 12 years, and tables for converting raw scoresto standard scores are provided in the test manual. Scores of normaladults approach ceiling levels. Research by Abbeduto, Furman & Davies(1989) has demonstrated that the test is suitable for American as wellas British subjects.

Curtiss and Yamada Comprehensive Language Evaluation (CYCLE) (Curtissand Yamada, 1980): CYCLE is an instrument which can be used to assessreceptive and expressive language in a comprehensive manner that farsurpasses any previously available language assessment battery. Thoughnot available yet commercially, research data have been collected onover 70 normal children at ages 2 through 8 years, and 100 languageimpaired children at ages 4 through 8 years. Considerable reliabilityand validity data on the test are available. CYCLE includes threebatteries: 1) Receptive, 2) Elicitation and 3) Free speech analysis.Only the receptive battery (CYCLE-R) was used in this study.

Items from the CYCLE-R were used to test receptive knowledge ofphonology, lexical and relational semantics, inflectional andgrammatical morphology, and syntax. The test is comprised of a set ofitems which cover the age range of 2-9 in each linguistic area mentionedabove. In scope of language areas and age range covered, it is the testmost ideally suited to this study. Each item on the test includesvocabulary that is easily picturable and familiar to young preschoolchildren. Each item also, when possible, tests only one grammaticalelement or structure, so that comprehension of each aspect of thegrammar can be ascertained separately. Since knowledge of complexstructures entails knowledge of the component simple structuresinvolved, all test items are developmentally arranged, and more complexitems are administered only after a child has passed the items precedingit in level. Every item involves five example sentences; thus eachstructure is tested enough times to establish performance consistency,and there is some insurance that chance alone will not allow a child toperform well. Each example sentence has enough choices in the responsearray to further ensure that a child cannot reach a passing criterion onan item by chance.

The CYCLE-R incorporates several features which make it a well-designedreceptive measure for testing young children. Each item involvingpictures uses decoys consisting of the identical elements mentioned inthe test sentence, but in a different relationship from that of thecorrect test picture. In addition, decoys which allow the examiner todetermine if the child has attended to only the first or last part ofthe test sentence are also included. Extra linguistic and redundantlinguistic cues have been eliminated from all items. Thus there are nocues in the test array or in the test sentences themselves that aid testperformance. All pictures are clear, simple line drawings which directlyrepresent the relevant information; i.e., no interpretative load isplaced on the child. In addition, there is a clear separation betweenall pictures in an array, and all illustrations are large enough forchildren to see and interpret. There is never a time when more than onechoice in a picture array is tested, so that process of eliminationcannot be used to aid performance. Correct choice as well as all decoychoices are randomized throughout an item; e.g., correct choice occursin each array position equally often. Each distinction is tested by morethan one token or lexical item reflecting that distinction. CYCLEperformance yields a raw score which can be converted to age equivalentscores based on normative data.

Goldman-Fristoe-Woodcock (GFW) Test of Auditory Discrimination (Goldman,Fristoe, and Woodcock, 1970): This test was designed to assessspeech-sound discrimination of single syllable words under controlledlistening conditions for children 3 years 8 months through adults. Thetest consists of two parts: 1) pretest to determine knowledge of testvocabulary; 2) auditory discrimination of single words through picturepointing format. The subject listens to a word presented on a taperecorder (for example lake) and must select from four pictures (lake,rake, wake, snake) the picture representing that word. All minimal pairspeech sound contrasts are assessed for discrimination in this test.

Expressive Language Tests

Goldman-Fristoe Test of Articulation (Goldman & Fristoe, 1986): TheGoldman-Fristoe Test of Articulation provides a systematic andcomprehensive measure of a child's ability to articulate the consonantsounds present in speech. Its results provide a measure of all thenecessary phonemes and determine which sounds are produced incorrectly,as well as the type of misproduction. The sounds in words subtest whichuses a series of pictures to elicit the major speech sounds in theinitial, middle or final position was administered to each child. Thetest manual provides normative data for ages 2 through 16 years.

Memory for Sentences Subtest (Stanford-Binet Intelligence Scale;Thorndike, Hagen, & Sattler, 1986): This subtest measures the ability torecall meaningful sentences. The subject hears each sentence and isrequired to immediately reproduce the sentence verbatim beforecontinuing. Each sentence is scored Pass/Fail and testing isdiscontinued when three of four sentences are failed. Sentences increasein syllable length and syntactic complexity with increasing items.Normative data are provided for ages 2 through 23 years.

CELF--Recalling Sentences Subtest (Semel & Wiig, 1980): TheComprehensive Evaluation of Language Functions (CELF) language batteryhas been standardized on a large sample of children in the U.S.A. agedfrom 6 to 16 years. After a demonstration item and a practice item, thesubject is required to repeat verbatim a series of 26 sentences ofincreasing length and complexity, and the response is awarded threepoints for correct repetition, 2 points if there is one error, 1 pointif there are 2 to 3 errors, and zero points for 4 or more errors.Testing is discontinued after 4 consecutive zero scores. The test manualprovides tables for converting scores to scaled scores.

Training Sessions

Eleven training sessions were developed specifically for this study.Sessions were given in twenty minute periods in random order accordingto the schedule described above. Two sections (sessions A and B) usecomputer game formats already described in detail as embodiments of thisinvention (see above) and so will only be described briefly here. Theother sessions (sessions C through K) use taped or CD-ROM speech andlanguage segments in which the acoustic waveform was computer expandedand enhanced using the methods described in detail as embodiments ofthis invention (see below).

Session A--Nonverbal Temporal Sequencing Training

An interactive multimedia game mounted on a CD-ROM (Circus SequenceGame) was used to train temporal order sequencing of nonverbal sounds(e.g. frequency modulated sweeps and steady state tonal complexes) atfaster and faster presentation rates. The nonverbal stimuli wereconstructed specifically to represent components of human speech. Forexample, the rate changes, the starting and ending frequencies and thestimulus duration of the frequency modulated sweeps were similar tosingle consonant formants in normal human speech. Similarly, the steadystate tonal complexes were constructed to mimic vowels in human speech.Subjects were trained to listen to a 2-element stimulus sequence, whichwas selected at random from four possible 2-element stimulus sequencecombinations, in which the stimulus elements were separated by a briefinterstimulus interval (ISI) (see FIG. 2). Subjects were required torespond by pushing panels on the touch screen of a computer to indicatethe order of stimulus presentation (see FIG. 3). Correct responses werereinforced by a presentation of a single brief sound that was alwaysassociated with a correct response, earning a point which was displayedas a cumulative score on the video screen. In addition, when threecorrect responses in a row were obtained, this string of correctresponses was rewarded by a brief animation selected from a large poolof animation sequences. The high quality and wide variety of animationsserved to maintain interest in playing the game. The program itself usedan adaptive tracking procedure in order to determine the interstimulusinterval for subsequent stimulus sequence presentations. That is, as thesubjects succeeded at identifying the stimulus sequence order, the taskwas made progressively more difficult by reducing the interval betweenthe two stimuli.

Session B--Syllable Discrimination

An interactive multimedia game mounted on a CD-ROM (Phoneme Clown Game)was used to train syllable discrimination using minimal phoneme pairs.Both normal phoneme pairs and modified phoneme pairs where specificaspects of the acoustic signal had been computer modified to enhancetemporal cues were used in this training exercise. In addition, theinterstimulus interval between the two phoneme pairs was systematicallyvaried in duration. This provided experience and training at differentphoneme presentation rates. The computer screen displayed a circus scenethat included three clown faces. Initially, a single clown face appearedon the left of the computer screen. Subjects initiated each trial bytouching this clown face. Once touched, this clown said a specificsyllable (eg., /ba/) and the uppercase letter B appeared over theclown's face. Next, the left most clown disappeared and two new clownsappeared on the screen and two more syllables were presented one afterthe other. The subject's task was to determine whether the first orsecond syllable matched the sound and letter produced by the firstclown. L/LD children indicated their response by touching the first orsecond clown which appeared on the right side of the computer screen.Correct responses caused the target letter (e.g. B) to appear over theface of the touched clown and were rewarded by a wide variety ofcomputer animations, "reward sounds", and points. A cumulative pointscore was displayed on the computer screen. If incorrect, no reward wasgiven and the subject proceeded to the next trial. In addition, aftercompletion of each 50 blocks of trials a brief video clip from an ageappropriate animated movie appeared embedded in the game screen. Thesevideo clips were sequential so that continued practice resulted inpresentation of the next segment of the animated movie.

Session C (Listening to Stories)

Popular storybooks for children (eg. Disney's and Seuss books) were taperecorded on a digital audio tape recorder (i.e. DAT). Next, the digitalaudio tape was transferred to a computer in digital form so thatelongation and enhancement of certain portions of the speech waveformcould be performed, as described in the embodiment of the inventionsections above. Alternatively, commercially available interactivemultimedia CD-ROM children's stories were used. The digital audioportions of the CD-ROMs were extracted and transferred to a computer sothey could be processed as described above. Once modified, the audiosignals were converted back into analog tape for the "books on tape"versions or reinserted in the appropriate digital formats for the CD-ROMstories and new CD-ROMs containing the modified speech were produced. Itwas these speech modified "books on tape" and CD-ROM based stories thatwere used in this training session. Subjects selected a story of theirchoice and listened to them during at least one 20 minute session perday at the laboratory, either on tape with an accompanying book or onCD-ROM. Each day, each child checked out tapes and accompanying books orCD-ROM based stories from the laboratory "library" to take home tolisten to as homework for one to two hours per night. The materials werereturned the next day or after the weekend and new stories were selectedfor the next night or weekend's homework allowing the stories to rotatebetween the children, increasing the variety available. Subjectsreceived 50 points for each story they listened to. Parents kept trackof these points for homework sessions so they could be added to thechild's weekly points score.

Session D--Receptive Grammar Training--(CYCLE Format)

The CYCLE format was used for training specific elements of receptivegrammar. For each item of the original CYCLE (form A) a new item wasselected in order to develop a new series of items to be used intraining (form B). Items were selected so that vocabulary and contentwere changed but the essential semantic, syntactic and morphologicalelements remain the same. Take for example the sentence "the fish areswimming" which appears on the CYCLE form A. The child must selectbetween two pictures one which shows a single fish swimming and theother which shows several fish swimming. To respond correctly theindividual must know that in this case the verb carries the pluralinformation and therefore the only correct answer is the picture withseveral fish. This differs from most other forms of pluralization suchas occur in the sentence "the dogs are running" where information aboutpluralization occurs both in the noun "dogs" as well as the verb "are".For the training component, an equivalent sentence was developed inwhich pluralization was carried solely by the verb "the deer are eating"which appeared on form B. All items were similarly converted to maintainthe grammatical component essential to comprehension while altering thevocabulary, thus assuring that not merely training knowledge of aspecific item was achieved but rather grammatical rules were alsofollowed. Form B items were tape recorded and subsequently subjected tocomputer modification of the acoustic waveform using the proceduresdescribed above.

During training, subjects were asked to point to the picture thatrepresented each command. They received immediate feedback indicatingcorrect or incorrect responses (i.e.,"thumbs up" or "thumbs down").Regardless of whether the response was correct or not, the command wasrepeated again so that they could listen to it directly after theexperimenter had indicated which was the correct picture. Subjectsreceived a sticker to be placed on a wall chart for completing thissession.

Session E--Receptive Memory Training (TOKEN Format)

The TOKEN test format was used for training receptive memory. Using thesame procedure as described above, items from form A of the TOKEN testwere modified to develop a new set of items which made up form B. Forexample, the item "touch the large red circle and the small greensquare", which appears on form A of the TOKEN test, was changed to"touch the small yellow square and the large blue circle" for form B.The new form B items were tape recorded and computer modified in thesame manner described above. Training included two repetitions of eachitem with feedback, also as described above. Subjects received a stickerto be placed on a wall chart for completing this session.

Session F--Receptive Grammar Training (TROG Format)

The TROG test is similar in format to the CYCLE. The identical procedurewas followed for developing a form B of this measure and computermodifying the acoustics for use in training as described above. Thetraining procedure and reinforcement was the same as that used for theCYCLE and TOKEN session.

Session G--Receptive Phonology Training (GFW Format)

The GFW format was used to train receptive phonology. Here the subjectlistens to a word presented via a tape recorder and points to thepicture that represents that word, amongst other pictures whichrepresent words which differ by only one distinctive feature of speech.Using a similar procedure to that described above, an alternate form(form B) was developed for training receptive phonology. The new itemswere tape recorded and modified accordingly for presentation during thetraining sessions. Once again, subjects received each item twice withthe second repetition being given with prior knowledge of the correctresponse. Subjects received a sticker for reinforcement.

Session H--Sentence Imitation Training (CELF Format)

Items from the CELF sentence imitation subtest were modified in a mannersimilar to that described above to develop items for form B. However,for these items, specific and careful attention was paid to selectingwords which had equivalent frequencies in the language and roughlysimilar phonological complexity. Form B items were then tape recordedand modified in the manner described above. Subjects were presentedthese new modified sentences and asked to repeat them verbatim. Nofeedback was given. All errors, including phonological productionerrors, were scored. The CELF includes both grammatically correct andgrammatically incorrect sentences for repetition. Separate scores foreach of these types of items are analyzed.

Session I--Expressive Memory Training (Memory for Sentence Format)

The Memory for Sentence subtest of the Stanford-Binet was used as amodel for developing training items for this session. This session isidentical to session H above (CELF) except that only grammaticallycorrect sentences are used. The procedures followed for itemdevelopment, stimulus modification and testing procedures are identicalto those described above.

Session J--Repeating Real and Nonsense Syllables and Words.

A list of nonsense syllables and words were developed specifically forthis study. Careful attention was placed on acoustic and phoneticcharacteristics of the stimulus set. The goal of this training was todetermine whether acoustic alteration of the input would result inimproved articulatory coding of speech output at the acoustic, phoneticand phonological levels of analysis. It was of particular interest todetermine whether training, using materials in which the acoustictemporal cues had been altered, would result in improved control ofthese same temporal cues in speech production.

Session K--Receptive Grammar Training (Simon Says Format)

A Simon Says game format was used to train receptive grammar and memory.Items were developed specifically for this training which were similarto those appearing in several of the other receptive language trainingsessions. However, in this format, instead of responding by selectingappropriate pictures or pointing to static circles and squares, subjectswere required to act out each command using a series of props. Trainingitems were developed and tape recorded and subsequently computermodified using the same acoustic modification described above. Someitems began with "Simon Says" which was the child's instruction to doexactly what Simon said to do. To enhance the fun of the game, itemswere inserted without the prephrase "Simon Says". These were followed byphrases such as "don't let me catch you, wait for Simon". Items werespecifically constructed requiring individual phonological contrast (eg."touch your nose" vs. "touch your toes") or grammatical contrast (suchas, "hop on your foot" vs. "hop on your feet"). Sequential commands werealso included, such as "touch your nose, shoulders, toes and knees".Other items were constructed to train receptive memory, such as "runaround the large blue crayon and then pick-up the small yellow spoon"(where large and small crayons and spoons of various colors wereavailable as props). Some of these commands were proceeded by "SimonSays" while others were not. Once again, as was the case with each ofthe other receptive language training sessions, each command wasrepeated twice with feedback given after the child's response todemonstrate the correct action if the child was incorrect. This wasfollowed by a second opportunity to hear and act out the command withprior knowledge of the correct response.

Reinforcement

In order to maintain motivation throughout the program, a series ofpoint and sticker based reinforcement systems were developed. Thecomputer games used each day in sessions A and B had a points andsticker won ongoing total built into the program which were displayed onthe screen in a cumulative fashion after each response. In addition,intermittent computer animations and movie clips were used to maintaininterest in these training games. The points and stickers won by playingthe computer games were transferred at the end of each day, as a groupactivity, to a decorated wall chart that tracked each child'saccumulations week by week. At the end of each week, the children coulduse these points to "buy" toys from the "Circus Store". The storecontained toys marked with "prices" which corresponded to numbers ofpoints earned.

In addition to these weekly prizes the children earned stickers forcompleting each of the other 20 minute training sessions. Unlike thepoint system used in the computer game, however, these were notperformance based but rather indicated completion of a session. These"special stickers" were placed on a chart so the child could monitortheir progress through these training sessions each week. Once a chartwas filled (indicating that the child had completed the requisite numberof sessions for that week) they could select a toy or food item from the"Listening Store" or "Speaking Store". These reinforcers, together withpersonal attention and reinforcement by the experimenters, proved highlysuccessful in maintaining motivation throughout the six week program.

Results

Two training procedures were employed in this study. First, an attemptwas made to alter the highly impaired temporal thresholds that the L/LDchildren demonstrated in week 1 (pretest) by selectively trainingtemporal integration rates daily during weeks 2-5 using the CircusSequence CD-ROM game format described in the embodiment of theinvention. The Tallal Repetition Test was given in weeks 1 and 6 as thebenchmark measure used to assess change in temporal processingthresholds. As can be seen in FIG. 7A, at pretest (week 1) each childdemonstrated the ability to perform the two tone sequencing subtest ofthe Repetition Test using 150 msec duration tones with 0 msec ISI. Thisdemonstrates that each child understood the task and could respond at ahigh level of accuracy when sufficiently long duration stimuli werepresented. However, when the duration of the stimulus was reduced to 75,40 or 17 msec respectively, the L/LD children's performancedeteriorated. Recall that normal control 6-8 yr. old children onlyrequire 8 msec between two 75 msec tones. In contrast the L/LD childrenrequired an average of 221 msec. Similarly, the L/LD children showedprogressively greater deficits as the duration of stimulus presentationwas decreased to 40 or 17 msec.

The results following four weeks of training on the Circus Sequence gameare dramatic. As can be seen in the top of FIG. 7B (week 6 post-test)temporal processing thresholds for each child were substantiallyreduced. Group totals also reflect this marked improvement. For example,using 75 msec tone durations, the average ISI for the group went from221 msec at pretest to 32 msec at post test, a temporal processing ratemuch closer to the normal range of 8 msec. Similarly, the mean thresholdfor 40 msec duration tones went from 640 msec at pretest to the posttraining rate of 286 msec, again a highly significant improvement.

These data provide strong evidence of the efficacy of the temporaltraining procedure (Circus Sequence game), embodied in this invention,for the modification of temporal integration deficits in L/LD children.

The second training procedure developed in this invention utilizedcomputer extended and enhanced speech to train speech and languagefunctions. Of specific interest was whether the rate of development ofbasic phonological processes as well as higher level linguisticabilities (morphology, semantics, syntax) could be enhanced throughtraining with computer modified acoustic input. It was hypothesized thatthe acoustic modifications that were made in the fluent speech used fortraining would result in the greatest improvements in receptive aspectsof speech (phonology) and language comprehension. This hypothesis can bedirectly tested by comparing week 1 (pretest) to week 6 (post test)performance on each of the benchmark tests of receptive phonology andlanguage processing. In FIGS. 8 through 11, data are presented first foreach of the individual subjects participating in the study, followed bythe average or means for each of the benchmark receptive language tests.When available in the test manual, based on standardized normative datafor the test, age equivalent scores have been presented. When ageequivalents are not available, percentiles for age or raw scores arepresented.

As can be seen in FIGS. 8 through 11, remarkable improvement thedemonstrated for each of the receptive language and memory tests(Session D, E, F, G). The difference between pretest and post testscores are highly statistically significant, demonstrating markedimprovement in language reception, including phonological (GFW test)(FIG. 8) as well as grammatical understanding of language (TROG andCYCLE) (FIGS. 9 and 10) tests and memory for language (TOKEN test)abilities (FIG. 11). These data represent remarkably strikingimprovements in the core receptive language functions that cause thegreatest difficulty for L/LD children. Further, it is precisely theseabilities that have been shown in longitudinal outcome studies of L/LDchildren to be most predictive of their subsequent academic achievement.Indeed, improvement occurred within six weeks that is comparable to orsurpasses that found in a previously completed five year longitudinalstudy of L/LD children.

In the San Diego Longitudinal study (directed by Tallal and Curtiss,1980-1987) the language development of 60 L/LD children, selected basedon the same criteria used in the current study, was assessed over a fiveyear period. As many of the same benchmark measures were used in boththe previous longitudinal study and the current training study one canmake direct comparisons. FIG. 12 shows the results of improvement on theCYCLE test from the age of 4 to the age of 8 years for the L/LD childrenin the longitudinal study. The L/LD children were receiving speech,language and reading therapy within the public school for theirdisabilities during the time course of the longitudinal study.Therefore, one can compare directly the state-of-the-art therapiescurrently available to L/LD children in the public schools to thetherapy embodied in this invention. In the current study at pretest theL/LD children as a group (mean age=7.2 yrs.) were performing at a levelequivalent to 4.2 year old normally developing children on the CYCLEtest. However, after only 4 weeks of exposure to training with theacoustically modified speech stimuli, as well as temporal sequencingtraining using the computer games, these children were able to processlanguage at a level equivalent to 5.8 year old normally developingchildren. In contrast, results from the San Diego longitudinal studydemonstrate L/LD children progress very slowly in their development oflanguage comprehension taking 5 years to develop the equivalent skillsthat normally developing children acquire in 21/2 years. In strikingcontrast, the L/LD children participating in the current remediationstudy made very rapid progress. After only 4 weeks of training withcomputer modified speech and temporal integration training, theirdevelopment of core language comprehension abilities improved by 1.6yrs. whereas the longitudinal sample improved only 2.6 yrs. on the samemeasure over a 5 year period with conventional speech, language andreading therapies. Similar results are seen on the Token Test and GFW(FIGS. 13, 14).

Although computer alterations of the acoustic input of speech wasspecifically aimed at training the receptive aspect of language (fromthe phonological to the syntactic level), measures of speecharticulation and expressive language functioning were also collectedthroughout the study. As expected, less improvement was demonstratedfrom pre to post test on the expressive language measures, as these werenot explicitly trained in this study (see FIGS. 15, 16).

It is encouraging to note that a significant improvement in speecharticulation was accomplished at the phonological level even though thiswas not directly trained in this study. Previous research with L/LDchildren demonstrated that their temporal deficits occur in both theperceptual and motor systems affecting both speech input as well asoutput at the acoustic as well as phonological level. Thus, it can behypothesized that training in the rate of temporal integration at theperceptual level, as well as exposure to temporally modified speech,might not only improve perceptual processing and thus receptive languagecomprehension, but also the rate of motor precision in the production oftemporal cues in speech articulation. The significant improvement foundon the Goldman-Fristoe Articulation Test (FIG. 17) provides tentativesupport for this hypothesis at the phonological level. Strong supportawaits future detailed computer analyses of precise temporal motorplanning, at the acoustic and phonetic levels based on the speechproduction data obtained during this training study. However, thecurrent analysis at the phonological level demonstrates significantimprovement in speech articulation and suggests that these perceptualtraining measures may not only improve aspects of receptive language andreading, but may also have a direct impact on improving the temporalprecision of articulatory coding (and therefore intelligibility) of thespeech output of these children.

In conclusion, these data provide extremely strong support for theefficacy of the therapies embodied in this invention. It has beendemonstrated that these therapies significantly improve the temporalprocessing, speech, language and reading abilities of L/LD children andthat the magnitude of improvement is considerably greater than has beendemonstrated to result from currently available therapies.

Six of the seven children were available for post test assessment inweek 12 (graphed as A₃), after having received 6 weeks without exposureto modified speech or temporal processing training. As can be seen inthe previous figures as A₃ test results, although there was some declinein receptive language abilities when the therapy was discontinued, muchof the progress was maintained. These data suggest that a longertraining period may be necessary to fully remediate and "cement"fundamental speech, language and reading skills. Nonetheless, these dataare encouraging in that a substantial proportion of the progress thatwas made during this brief training period can be maintained even in theabsence of further direct training.

Thus, not only has this invention met the need for training individualswith a learning disability, it has also provided the capability to"cure" the temporal processing deficit that occurs in those individualswho are dysphasic or dyslexic, and are unable to process rapidlyreceived stop consonants, hence unable to grasp the meaning of thespoken words.

It is felt that the information set forth above with the experimentoutlined in great detail is a sufficient disclosure to enable one topractice this invention; hence, further discussion of its method ofoperation will not be undertaken.

This invention is not limited to the above description but is defined bythe following claims.

It is claimed:
 1. A computerized system for structuring recorded audioinformation to enable language-impaired individuals to better understandaudio sounds, the system comprising:a computer having input means forreceiving information including: a processor means for manipulating thereceived information; storage means for storing unprocessed receivedinformation and manipulated information; and output means responsive tothe processor means for presenting the manipulated information in a formunderstandable to a language-impaired user first program means formodifying recorded audio information which are digitally recorded audiosounds having a frequency range associated therewith to lengthenportions of selected phonemes without modifying the frequency range;second program means responsive to received information and saidprocessor means for storing the modified digitally recorded audiosounds; and, third program means responsive to received information andto said processor to direct said stored, modified digitally recordedaudio sounds to said output means.
 2. The system of claim 1 wherein saidfirst program means further includes means for amplifying said portionsof selected phonemes.
 3. The system of claim 2 wherein said thirdprogram means further includes means for simplifying the output signalby integrating said output sounds within a fixed number of band-passfrequency channels.
 4. The system of claim 3 wherein the third programmeans includes means to convert said stored, modified digitally recordedverbal sounds to aural signals.
 5. The system of claim 4 wherein theoutput means includes speaker means for generating an aural signal. 6.The system of claim 5 wherein the output means further includes a videodisplay tube and further wherein the third program means includesdisplay means for generating characters representing the aural signalsas the video displays on the video display tube coincident with thegeneration of aural signals, said video displays remaining on said videodisplay tube for a preselected time.
 7. A system of claim 5 the videodisplays related to temporally modified aural signals are subject toparallel selective temporal emphasis.
 8. The system of claim 1 whereinsaid output means generating an output signal comprising an output soundfrom said audio sound, and wherein said third program means furtherincludes means for simplifying said output signal by integrating saidoutput sounds within a fixed number of band-pass frequency channels. 9.The system of claim 1 wherein the third program means includes means toconvert said stored, modified digitally recorded audio sounds to auralsignals.
 10. The system of claim 9 wherein the output means includesspeaker means for outputting said aural signal.
 11. The system of claim10 wherein the output means further includes a video display tube andfurther wherein the third program means includes display means forgenerating characters representing the aural signals for display on thevideo display tube coincident with the generation of aural signals, saidcharacters remaining on said video display tube for a preselected time.12. A method of increasing the ability of a human being to processaurally received audio sounds, the method comprising:a) recording verbalsounds in a computerized system; b) modifying the recording of certainshort duration phonetic elements of said verbal sounds by computerprogram means without changing the fundamental frequencies of thoseelements and producing the digital signals from said verbal sounds; c)converting said modified digital signals to an analog signal; and, d)generating aural signals of the converted analog signal.
 13. The methodof claim 12 wherein the modifying step includes the step of emphasizingselectively the short-duration phonetic elements.
 14. The method ofclaim 12 wherein the modifying step includes the step of recognizingconsonant sounds for elongation by identifying the rapid andphoneme-specific changes of their frequencies and intensities.
 15. Themethod of claim 12 wherein the aurally received audio sounds are aplurality of spoken words in textual order, and wherein the modifyingstep includes the steps of elongating portions of selected phoneticelements.
 16. The method of claim 15 wherein the modifying step includesa shortening of vowel sounds associated with the elongated portions ofselected phonetic elements and shortening some silent intervals so thatthe elapsed time for generating the aural signals is substantially thesame as the elapsed time to record said verbal sounds.
 17. The method ofclaim 16 wherein the modifying step includes the step of emphasizingselectively the short-duration phonetic elements.